(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
International Bureau 




iiiiDiioiiiinniiiiiiHiiHii 



(43) International Publication Date (10) International Publication Number 

10 May 2002 (10.05.2002) PCT WO 02/37421 Al 



(51) International Patent Classification 7 : G06T 1 1/00 

(21) International Application Number: PCT/EPO1/06743 

(22) International Filing Date: 13 June 2001 (13.06.2001) 

(25) Filing Language: English 

(26) Publication Language: English 

(30) Priority Data: 

60/245,507 3 November 2000 (03.1 1.2000) US 

(71) Applicant (forAE, AG, AU, BB, BZ, CA, CY, GB, GD, GH, 
GM, IE, IL, KE, LC, LK, LS, MN, MW, NZ, SD, SG, SL, SZ, 
TT, TZ, UG, ZA, ZWonly): UNILEVER PLC [GB/GB]; 
Unilever House, Blackfriars, London EC4P 4BQ (GB). 

j= (71) Applicant (for A I, AM, AT, AZ, BA, BE, BF, BG, BJ, BR, 
= BY, CF, CG, CH, CI, CM, CN, CO, CR, CU, CZ, DE, DK, 
= DM, DZ, EC, EE, ES, FI, FR, GA, GE, GN, GR, GW, HR, 
= HU, ID, IS, IT, JP, KG, KP, KR, KZ, LR, IT, LU, LV, MA, 
= MC, MD, MG, MK, ML, MR, MX, MZ, NE, NL, NO, PL, 
= PT, RO, RU, SE, SI, SK, SN, TD, TG, TJ, TM, TR, UA, 
III UZ, VN, YUonly): UNILEVER NV [NL/NL]; Weena455, 
== NL-3013 AL Rotterdam (NL). 

= (71) Applicant (for IN only): HINDUSTAN LEVER LIM- 
ITED [IN/IN] ; Hindustan Lever House, 1 65/1 66 Backbay 
Reclamation, Maharashtra, 4O0 020 Mumbai (IN). 



(72) Inventors: HAWKINS, Stacy, Susan; Unilever Research 
U.S. Inc., 45 River Road, Edgewater, NJ 07020 (US). AN- 
DREW, Jeremy, James; Unilever Research Port Sunlight, 
Quarry Road East, Bebington, Wirral, Merseyside CH63 
3JW (GB). MURAHATA, Richard, Iwao; Unilever Re- 
search U.S. Inc., 45 River Road, Edgewater, NJ 07020 
(US). 

(74) Agents: ELLIOTT, Peter, William et al.; Unilever PLC, 
Patent Department, Colworth House, Shambrook, Bedford, 
Bedfordshire MK44 1LQ (GB). 

(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, FI, GB, GD, GE, GH, 
GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, 
SL, TJ, TM, TR, TT, TZ, UA, UG, UZ, VN, YU, ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, 
IT, LU, MC, NL, PT, SE, TR), OAPI patent (BF, BJ, CF, 
CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— with international search report 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



< 



^ (54) Title: METHOD OF EVALUATING COSMETIC PRODUCTS ON A CONSUMER 

^5 (57) Abstract: A method and kit is provided for demonstrating the effects of a cosmetic product on a consumer's body, especially 
2 effects over a period of time. The method includes applying a cosmetic product to the body, capturing an image of the consumer, 
displaying the image on a monitor and digitally transforming the image to reflect the effect of using the selected cosmetic over the 
Q period of time. Among transformed attributes of the body are those of glow/color, sags/wrinkles, pores and combinations thereof. 
The original and transformed images arc juxtaposed on a screen. Consumers are required to select between transformed and dis- 
played images, preferably repetitively, until the consumer has chosen their most appealing transformation. 
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METHOD OF EVALUATING COSMETIC PRODUCTS ON A CONSUMER 

The present invention relates to a method for transforming 
5 an image of a consumer's skin, in particular facial skin, to 
an image with a consumer-defined composite cosmetic 
attribute. 

Cosmetic companies have developed on-counter visualization 
10 tools to show wrinkles and how they can worsen, based on 
increasing the length and width of the wrinkles. These tools 
are not based on actual consumer or clinical data, or actual 
studies of the progression of a wrinkle over time. In 
addition, many companies have the technology to display 
15 actual images with color changes to the face, without showing 
how the face would appear as a result of improving a 
consumer-defined composite cosmetic attribute. See for 
instance JP 7100126 and JP 5196436 (Shiseido) . Another 
patent from Shiseido (JP 63006461) provides recommendations 
20 from a video editing system based on the roughness of a 
replica taken on counter. 

JP 8299288 (Kanebo) describes "spot" measurements, which may 
be related to dryness, oiliness, or color over a small (mm 
25 size) region on the skin. While these measurements are on- 
counter diagnostics, they do not include an overall attribute 
visualization of the face as in the present invention. 

Clinique has a web site for product recommendation, based on 
30 user input. It does not involve actual changes or expert 
ratings in any composite cosmetic attribute or healthy skin. 
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Revlon in U.S. Patent 5,163,010 describes a similar system 
for recommending custom mixed products based on surveys at 
the point of sale. 

5 Studies exist on the relationship of facial features to the 
perception of age and gender. See for instance Hawkins SS, 
Perrett DI, Burt DM, Rowland DA and Murahata RI. * Prototypes 
of facial attributes developed through image averaging 
techniques", Intern. J. of Cos. Science, 21; 159-166 (1991); 

10 Rowland DA, Perrett DI, Burt DM, Lee KJ and Akamatsu S. 
"Transforming Facial Images in 2 and 3-D", Imagina 97- 
Conferences-Actes/Proceedings, Pages 1-8, February 1997; 
Burt DM, Perrett DI. "Perceptual asymmetries in judgments 
of facial attractiveness, age, gender, speech and 

15 expression" Neuropsychologia 35 (5) : 685-693, 1997; Burt DM, 
Perrett DI. "Perception of age in adult Caucasian male 
faces: computer graphic manipulation of shape and colour 
information." Proc. R. Soc. Lond. B 259:137-143, 1995; 
Rowland DA, Perrett DI. "Manipulating facial appearance 

20 through shape and color." IEEE Computer Graphics and 
Applications 15(5):70-76, 1995; Perrett DI, May KA, 
Yoshikawa S. "Facial shape and judgments of female 
attractiveness." Nature 368:239-242, 1994. These articles 
have advanced imaging technology but have not been applied 

25 specifically to model individual attributes, which taken 
collectively define a composite cosmetic attribute (e.g., 
"healthy" skin), sought by a consumer. 

Accordingly, the present invention aims to provide a method 
30 for presenting an image of a consumer's skin, especially 
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that of the face, in a manner reflecting a cosmetic 
product's effect over a period of time. 

The present invention also aims to provide a method for 
5 evaluating a cosmetic product on a consumer' s face or other 
body parts through image transformation over a period of 
time. 

In another embodiment the present invention aims to provide 
10 a method wherein an image of an individual's actual face is 
transformed over time and the effects of a cosmetic product 
on color, sags, wrinkles, texture and radiance can be 
demonstrated. 

15 In yet another embodiment the present invention aims to 
provide a method for demonstrating to a potential customer 
the benefit of certain cosmetic products when used over a 
period of time and allowing comparison with the effects of 
non-treatment. 

20 

These and other aims of the present invention will become 
more fully apparent from consideration of the following 
summary and detailed discussion. 
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Accordingly, a method is provided for demonstrating the 
effectiveness of a cosmetic product on a consumer's body 
parts when applied over a period of time, the method 
comprising: 

5 (i) selecting a cosmetic product for trial on the 

consumer's body and identifying a body feature to 
be observed over the time; 

(ii) capturing an image of the body part; 

(iii) displaying the image on a monitor; 

10 (iv) digitally transforming the selected body feature 

on the displayed image in conformance with a 
predicted effect of the cosmetic product and 
displaying the digitally transformed image; 

(v) allowing the consumer to compare results of the 
15 transformed versus the displayed image; and 

(vi) optionally repeating steps (iv) and (v) until the 
consumer has chosen an optimal transformation. 

Among the body features which may be manipulated are those 
20 of texture (pores), sags/lines/wrinkles and radiance/color. 
In most instances the body part selected for transformation 
will be that of the face or portions of the face. 
Comparison of images in step (v) may be tiled on the same 
screen. Alternatively, the displayed and transformed images 
2 5 may be on separate screens. They may also be viewable on 
consecutive screens, usually by a single click of a computer 
mouse or key. The tile option is preferred. This mode may 
also include having more than one transformed image placed 
on a screen with the original displayed image. For 
30 instance, three transformed images may be placed adjacent to 
a displayed image wherein each of the transformed images 
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represents the effects of using a particular cosmetic 
product over one month, two months and one year. 

The present invention is also concerned with a method for 
5 identifying the cosmetic needs of a consumer with respect to 
physical changes that are personal to the consumer, the 
method comprising: 

(i) gathering information from the consumer 
concerning the consumer' s cosmetic needs in 

10 treating a body feature; 

(ii) capturing an image of the consumer's body 
feature; 

(iii) displaying the image on a monitor; 

(iv) digitally transforming in a manner reflecting the 
15 effect of a recommended cosmetic over a period of 

time on a property of the body feature selected 
from color, sags, wrinkles, texture, radiance and 
combinations thereof, and displaying the 
transformed image; 
20 (v) requiring the consumer to select between 

transformed and displayed images; and 
(vi) optionally repeating steps (iv) and (v) until the 
consumer chooses an optimal transformation. 

25 A further aspect of the invention there is provided a kit 
for displaying .a consumer's skin image on a computer screen 
and transforming the image into consumer-defined composite 
attributes. Included is a camera for obtaining a high 
resolution image of the consumer's skin or hair, a screen 

30 for reeiving and displaying the image and a programmable 
device, including a memory program, for receiving the image, 
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transforming the image to an optimum image by subtracting 
from or adding to the image a number of difference images 
constructed from prototypes for each attribute of a 
composite cosmetic attribute. 

In a preferred embodiment, the programmable device also 
evaluates the direction and amount of the change for each 
attribute of the composite cosmetic attribute, so that a skin 
cosmetic product or regimen can be recommended to a consumer. 
Recommendation may also be done by a programmable device. 

When the method focuses upon aging attributes, improvement 
from one image to another can be expressed as a percentage 
improvement. According to the present invention, it is 
possible to replace the percentage concept simply with a 
value of number of years reduction in apparent appearance. 

The invention also * provides a method for obtaining a 
transformation of a composite cosmetic attribute, the method 
comprising: 

(a) identifying, preferably via a consumer study, 
individual attributes of a composite cosmetic 
attribute and grading a plurality of subjects on 
each attribute; 

(b) computing an average for subjects rated low and an 
average for subjects rated high to obtain the high 
prototype and the low prototype for each 
attribute; 

(c) computing a difference image for the individual 
attribute between a high and a low prototype; 
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(d) obtaining an optimum image by subtracting from or 
adding to an actual captured image increments of 
the difference image. 

Preferably, an optimum image is obtained by simultaneously 
subtracting/adding difference images for several or all 
attributes that are identified. Thus the captured image is 
transformed by a consumer to an optimum image with a 
composite cosmetic attribute, to a degree desired by a 
consumer. The inventive device and method allow consumers to 
regulate the degree to which they. adjust each component of a 
composite cosmetic attribute. In a most preferred embodiment 
of the invention, the method is consumer-driven: first, the 
composite cosmetic attribute is defined by a consumer 
(preferably, by a large consumer panel) ; then prototypes are 
obtained from consumer gradings (preferably, expert panel 
grading of a large number of consumers) , and finally a 
consumer defines her/his image with a composite cosmetic 
attribute, being able to adjust each individual component of 
a composite cosmetic attribute. 

In another preferred embodiment of the invention, the 
transformed image defined by a consumer is subsequently 
analyzed by a cosmetics manufacturer (or a sales 
representative) to evaluate the amount and direction of the 
change for each attribute of the composite cosmetic 
attribute. Subsequently, a cosmetics manufacturer (or sales 
representative thereof) may recommend a skin cosmetic product 
or a skin cosmetic regimen which would assist the consumer in 
attaining the transformed image. 
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The inventive method and kit are capable of displaying a 
transformed image which contains an attribute not present in 
the captured image. For instance, a consumer may see glow in 
the transformed image, even if there was no glow at all in 
5 the captured image. 

Yet a further aspect of the present invention involves a 
method for identifying the progress in the effect of an 
applied cosmetic product upon a consumer which effect is 
10 personal to that consumer, the method including: 

(i) selecting a cosmetic product for trial on the 
consumer's body and identifying a body feature 
being observed over a period of time; 

(ii) capturing an image of the body feature; 
15 (iii) displaying the image on a monitor; 

(iv) capturing a further image of the body feature at 
a later time after the product has been applied 
by the consumer; 

(v) allowing the consumer to compare results between 
20 the captured and later time captured images; and 

(vi) optionally digitally transforming the captured 
image of step (ii) in conformance with a 
predicted effect of the cosmetic product and 
displaying the digitally transformed image 

25 thereby allowing comparison to the later time 

captured image representing an actual cosmetic 
product treated body feature as obtained from 
step (iv) . 

Now it has been found that a consumer' s actual face or other 
30 body part can be digitally ' transformed through a computer 
program to demonstrate the effects of a cosmetic product 
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treatment. The treatment may simply be application of 
different colors onto skin (such as evaluating the different 
shades of lipstick, blush, eye or lip liners, foundation and 
mascara) or even hair. The latter may include coloring, 
5 hair growth or even depilatation. Transformations of the 
present invention take into account not only topography and 
shape of a face but also its color and texture. An 
important aspect of the invention is that transformed and 
originally displayed images are placed either beside one 
10 another or on adjoining screen views. The consumer is 
forced to pick between two of the images. Forced choice 
ordinarily is between a transformed and an original 
displayed image but' may also occur between different 
transformed images. 

15 

According to the present invention a consumer is assisted in 
evaluating cosmetic products with a future predictive- 
transformation obtainable in the following manner. High 
resolution images are captured of body sites which may 
20 benefit from the cosmetic products. Feature points are then 
semi-automatically delineated along the skin surface of those 
sites. Images have been grouped by one or more of a series 
of attribute scales to define an Attribute S. 

25 Based on consumer perception studies, it was determined that 
female cons.umers wanted healthy looking skin. "Health/' skin 
is a composite cosmetic attribute, which has not been 
previously defined. The term "skin" as used herein includes 
skin on the face, neck, hands, chest, arms and legs. 

30 Attribute S was developed using a set of scales to define 
healthy skin. 
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Qualitative language generation with Caucasian females in the 
US was performed to determine the individual attributes of 
healthy looking facial skin, and expert graders were trained 
based on the consumer language. 

5 

The study classified various cosmetic facial attributes as 
consumer-perceivable attributes that contribute to the 
overall appearance of cosmetic health. Typical attributes 
included lines and wrinkles, pores, blemishes, softness, 
10 silkiness and glow/color. Thus, the individual indicia of 
healthy facial skin were defined objectively and by a 
consumer. 

Three attributes (glow/color, lines and wrinkles, and pores) 

15 were then worked on as follows: Expert evaluations and 
photographs were taken of 80 panelists with no 
treatment /product use. The panelists all wore white hair 
caps, black bibs, and had cleansed their faces with their 
normal cleansing products 2 hours prior to their arrival at 

20 the study site. The photographs were recorded to PhotoCD for 
processing. From these images, facial averages were computed 
for panelists with the 9 lowest ratings for healthy 
glow/color, and the highest ratings for lines and wrinkles, 
and pores. The methods for computing facial averages have 

25 been previously described in the literature. These facial 
averages, or prototypes, provided insight into the primary 
regions of interest for each of the three attributes. 
Difference images were than constructed by subtracting the 
low prototype from the high prototype. This procedure was 

30 repeated for 19 attributes in total, which together defined a 
composite cosmetic attribute "healthy skin." 
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Novice graders were prompted to "make faces look healthy," 
using varying percentages of difference images to apply a 
positive or negative transformation of a particular 
attribute. Subjects were shown 9 faces in each of the three 
5 attribute categories (glow/color, pores, and lines and 
wrinkles) . The transformations were shown in random order 
with counterbalanced directions of transformations and mid- 
range points. Three faces were rated high, three mid-range, 
and three low for each category by expert graders. It was 

10 confirmed that novice graders were able to describe what they 
were seeing in the process of transforming the image and that 
their description . corresponded well to the attributes 
developed by expert graders. Known physiological processes 
were detected on individual attributes, e.g. aging for 

15 lines/wrinkles. 

The novice graders recognized all of the transformations as 
changing perceived facial health in the transformed attribute 
along the correct transformation direction. For example, 
20 subtraction of an attribute resulted in a decreased 
perception of healthy glow, pores, or lines and wrinkles. 

The same experiment performed with novice graders was 
repeated with panel experts. Expert graders were, able to 
25 identify the attribute that was used in obtaining a 
transformed image. Like novice graders, expert graders were 
able to identify the correct direction for an attribute 
resulting in a transformed image. 

30 According to the present invention, a consumer has their 
picture taken at a sales counter. By adjusting the captured 
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image through varying amounts and directions of difference 
images, an operator obtains a consumer's definition of their 
own ideal healthy skin. Customized product information can 
then be given to the consumer at the counter to help achieve 
5 this ideal. Clinically proven changes in products could also 
be shown to the consumer on their own image using this point 
of purchase interactive imaging system. Instead of a retail 
counter, the whole process may be conducted from a consumer' s 
home via Internet transmissions and by use of a consumer's 
10 own digital camera. 

It should be understood that the specific forms of the 
invention herein illustrated and described are intended to be 
representative only. Changes, including but not limited to 
15 those suggested in this specification, may be made in the 
illustrated embodiments without departing from the clear 
teachings of the disclosure. Accordingly, reference should 
be made to the following appended claims in determining the 
full scope of the invention. 

20 
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CLAIMS 



1. A method for demonstrating the effectiveness of a 
cosmetic product on a consumer's body parts when 
5 applied over a period of time, the method comprising: 

(i) selecting a cosmetic product for trial on the 
consumer's body and identifying a body feature to 
be observed over the time; 

(ii) capturing an image of the body part; 
10 (iii) displaying the image on a monitor; 

(iv) digitally transforming the selected body feature 
on the displayed image in conformance with a 
predicted effect of the cosmetic product and 
displaying the digitally transformed image; 
15 (v) allowing the consumer to compare results of the 

transformed versus the displayed image; and 
(vi) optionally repeating steps (iv) and (v) until the 
consumer has chosen an optimal transformation. 

20 2. A method according to claim 1 wherein the body feature 
being transformed is the face. 

3. A method according to claim 1 or claim 2 wherein the 
body feature being transformed is lines and wrinkles. 

25 

4 . A method according to claim 1 or claim 2 wherein the 
body feature transformed is glow and color. 



5. 

30 



A method according to claim 1 or claim 2 wherein the 
body feature is skin pores. 
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6. A method according to any of the preceding claims 
wherein the time period for observing the body feature 
is one month, two months or one year. 

1. A method for the identifying cosmetic needs of a 
consumer that are personal to the consumer, the method 
comprising: 

(i) gathering information from the consumer 
concerning the consumer' s cosmetic needs in 
treating a body feature; 

(ii) capturing an image of the consumer's body 
feature; 

(iii) displaying the image on a monitor; 

(iv) digitally transforming in a manner reflecting the 
effect of a recommended cosmetic over a period of 
time a property of the body feature selected from 
color, sags, wrinkles, texture, radiance and 
combinations thereof, and displaying the 
transformed image; 

(v) requiring the consumer to select between 
transformed and displaced images; and 

(vi) optionally repeating steps (iv) and (v) until the 
consumer chooses an optimal transformation. 

8. A kit for displaying a consumer's skin image and 
transforming the image into consumer-defined composite 
attributes, the kit comprising: 

(a) a camera for obtaining an image of the consumer's 
skin or hair; 

(b) a screen for receiving and displaying the image; 
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(c) a memory program for transforming the displayed 
image in a manner reflecting an effect achievable 
by application of a cosmetic product over a 
period of time and for displaying the transformed 
image in close . relationship to the non- 
transformed image. 

9. A method for obtaining a transformation of a composite 
cosmetic attribute, the method comprising: 

(a) identifying, preferably via a consumer study, 
individual attributes of a composite cosmetic 
attribute and grading a plurality of subjects on 
each attribute; 

(b) computing an average for subjects rated low and an 
average for subjects rated high to obtain the high 
prototype and the low prototype for each 
attribute; 

(c) computing a difference image for the individual 
attribute between a high and a low prototype; 

(d) obtaining an optimum image by subtracting from or 
adding to an actual captured image increments of 
the difference image. 

10. A method for identifying the progress of a cosmetic 
product treatment effect upon a consumer, the effect 
being personal to that consumer, the method comprising: 

(i) selecting a cosmetic product for trial on the 
consumer' s body and identifying a body feature 
being observed over a period of time; 

(ii) capturing an image of the body feature; 
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(iii) displaying the image on a monitor; 

(iv) capturing a further image of the body feature at 
a later time after the product has been applied 

' by the consumer; 

(v) allowing the consumer to compare results between 
the captured and later time captured images; and 

(vi) optionally digitally transforming the captured 
image of step (ii) in conformance with a 
predicted affect of . the cosmetic product and 
displaying the digitally transformed . image 
thereby allowing comparison to the later time 
captured image representing an actual cosmetic 
product treated body feature as obtained from 
step (iv) . 
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